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The  RC  ATO  (2004-2008),  a  joint  program  between  the  U.S.  Army  Tank  Automotive  Research, 
Development  and  Engineering  Center  (TARDEC)  and  the  Army  Research  Laboratory  (ARL),  has  the 
mission  of  developing  the  appropriate  tools,  techniques  and  autonomy  to  maximize  mounted  and 
dismounted  control  of  ground  unmanned  systems  and  optimize  Soldier-robot  and  robot-robot  teams.  This 
was  accomplished  through  the  development  and  testing  of  assisted  autonomy  and  situational  awareness 
solutions,  optimizing  Soldier-machine  interface  usability  across  varying  display  sizes,  control  devices  and 
dissimilar  robotic  systems  and  the  deployment  of  unique  multi-modal  control  techniques.  The  ATO 
concluded  its  Capstone  Experiment  and  demonstration  in  September  2008.  This  paper  will  detail  the 
technology  developed  and  utilized  under  the  program  as  well  as  highlight  Capstone  Experiment  results. 

INTRODUCTION 

General  Background 

As  the  emerging  technologies  of  the  Army’s  Future  Combat  System  (FCS)  are  introduced  to  the 
battlefield.  Soldiers  will  increasingly  face  new  challenges  in  workload  management.  A  shifting  force 
structure  will  bring  increasing  responsibilities  for  the  next  generation  Soldier,  who  will  be  tasked  with 
effectively  utilizing  and  protecting  robotic  assets  in  addition  to  performing  other  primary  missions.  Our 
overall  program  goal  is  to  understand  FIR1  issues  in  order  to  develop  technologies  and  mitigations  that 
enhance  FIRI  performance  in  future  combat  environments.  Tools,  techniques,  and  autonomy  are  being 
investigated  to  maximize  mounted  and  dismounted  control  of  ground  and  air  unmanned  systems  and 
optimize  Soldier-robot  and  robot-robot  ground  and  air  teams.  This  includes  the  development  of  a  scalable 
user  interface  for  robotic  control  that  maximizes  multi-functional  Soldier  performance  of  primary  mission 
tasks  while  minimizing  unique  training  requirements  by  optimizing  and  standardizing  required 
interactions  and  managing  workload  associated  with  the  control  of  unmanned  ground  and  air  systems. 

Unmanned  System  Mobility 

Unmanned  system  mobility  is  integral  to  unmanned  systems  in  a  multitude  of  missions  to  include 
Reconnaissance  Surveillance  and  Target  Acquisition  (RSTA)  and  convoys.  The  ability  to  accomplish 
military  missions  while  remaining  safe  (i.e.,  maintaining  secure  mobility,  McDowell,  Nunez,  Flutchins,  & 
Metcalfe,  2008),  requires  force  protection  measures  that  address  the  enemy  threat  and  protect  the  Soldier 

*U.S.  Army  Tank  Automotive  Research,  Development  and  Engineering  Center 

+U.S.  Army  Research  Laboratory,  Human  Research  and  Engineering  Directorate 

tDCS  Corporation 

**Alion  Science  and  Technology  MA  &  D  Operation  Research  Division 

1 

UNCLAS:  Dist  A.  Approved  for  public  release 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1 .  REPORT  DATE  2.  REPORT  TYPE 

23  JUL  2009  N/A 

3.  DATES  COVERED 

4.  TITLE  AND  SUBTITLE 

Robotics  Collaboration  Army  Technology  Objective  Capstone  Soldier 
Experiment:  Unmanned  System  Mobility 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

Jillyn  Alban;  Keryl  Cosenzo,  Ph.D.;  Tony  Johnson;  Shaun  Hutchins; 
Jason  Metcalfe,  Ph.D;  Erin  Capstick 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

US  Army  RDECOM-TARDEC  6501  Ell  Mile  Rd  Warren,  MI 
48397-5000 

8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 

20047RC 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM! S) 

TACOM/TARDEC 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

20047RC 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

Presented  at  AUVSI’s  Unmanned  Systems  North  America  2009, 10  - 13  August  2009,  Washington  DC, 

USA,  The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBIECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION 

18.  NUMBER  19a.  NAME  OF 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  S  AR 

unclassified  unclassified  unclassified 

20 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


without  sacrificing  overall  system  performance.  One  cost  effective,  technically  feasible  method  of 
increasing  force  protection  for  sustainment  convoys  is  the  use  of  autonomous  mobility  technologies. 
Autonomous  mobility  is  a  technology  that  enables  a  vehicle  to  maneuver  across  terrain  utilizing  onboard 
sensors  to  interpret  obstacles  or  hazards  and  plan  motion  paths  without  requiring  significant  human 
intervention.  The  development  and  advancement  of  this  technology  has  led  to  usage  of  robotic  vehicles 
in  Future  Force  concepts  (Shoemaker,  Bomstein,  Myers,  &  Brindle,  1998).  As  such,  the  purpose  of  the 
research  is  to  assess  aspects  of  autonomous  technologies  that  facilitate  performance  in  secure  mobile 
operations. 

If  well  implemented,  automation  of  mobility  functions  during  routine  convoy  operations  are 
expected  to  enable  Soldiers  to  increase  their  local  situational  awareness,  provide  additional  vehicle  rest, 
and/or  perform  other  necessary  duties.  At  a  glance,  the  driving  function  seems  simple:  put  hands  on 
wheel,  orient  vehicle  on  the  roadway,  maintain  adequate  speed  and  avoid  objects.  Yet,  the  appearance  of 
simplicity  is  deceptive.  In  fact,  depending  on  variations  in  dynamic  contextual  factors  associated  with 
both  the  environment  and  state  of  the  operator,  mobility  tasks  carried  out  in  advanced  platforms  may 
impose  considerable  demands  on  psychophysical  workload  and  thus,  precipitate  performance  detriments 
that  can  be  quite  dangerous.  For  example,  it  has  been  demonstrated  that  the  high  level  of  workload 
associated  with  driving  increases  the  possibility  of  mistakes  occurring  even  when  driving  is  the  only  task 
being  performed.  Indeed,  a  nontrivial  difference  between  civilian  and  military  driving  is  that  a  civilian 
driver  focuses  primarily  on  safety  and  mobility  whereas  the  military  driver  must  also  devote  attention  to 
security.  The  additional  requirement  to  conduct  security  scanning  tasks  while  on  the  move  can  place  extra 
task  demands  on  the  operator  (Wickens  &  Holland,  2000).  Ensuring  this  secure  mobility  function, 
defined  as  the  capability  of  the  Soldier-system  to  traverse  terrain  in  a  manner  that  meets  mission  demands 
while  sustaining  local  area  awareness  (McDowell  et  al,  2008),  is  essential  to  preserving  survivability, 
sustainability  and  lethality  of  US  military  forces.  This  investigation  is  aimed  at  identifying  the  impacts  of 
the  autonomous  technologies  on  Soldier-system  performance  involving  unmanned  ground  vehicle 
platforms. 

Autonomous  mobility  is  expected  to  radically  change  the  role  of  the  in  vehicle  driver  and  the 
unmanned  vehicle  operator;  rather  than  acting  as  an  intensive,  active  director  of  the  vehicle's  movements, 
the  Soldier  will  become  a  less  intensive  observer  and  supervisor  over  the  autonomous  mobility  system. 
The  autonomous  mobility  technologies  in  manned  and  unmanned  vehicles  may  potentially  increase 
overall  system  performance  by  reducing  vehicle  operator  demands  and  thus  allow  simultaneous 
management  of  other  high  value  assets  or  multitasking.  Wickens’  “multiple  resource  theory”  suggests 
that  concurrent  performance  of  multiple  high  workload  tasks  that  draw  from  the  same  limited  resource 
pool  (perceptual  and  cognitive)  can  lead  to  performance  decrements  due  to  the  non-availability  of 
workload  resources  (Wickens  &  Holland,  2000).  In  a  two-operator  versus  a  three-operator  manned  crew 
study,  it  was  found  that  a  single  operator  who  actively  scans  the  local  environment  while  driving 
experienced  greater  mental  workload  than  an  operator  only  tasked  with  commanding  the  vehicle.  In- 
vehicle  driving  technologies  such  as  anti-lock  brakes,  electronic  stability  control,  and  electric  power 
steering  have  been  used  by  the  civilian  automotive  industry  to  improve  vehicle  handling  and 
subsequently,  have  been  observed  to  increase  vehicle  safety  in  emergency  situations  where  a  human 
would  not  be  able  to  act  quickly  enough  to  avoid  dangerous  conditions  (Sakai,  Yoneda,  &  Shimizu, 

2004).  Other  related  technologies,  such  as  drive -by-wire,  already  have  shown  clear  potential  to  improve 
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driving  performance,  increase  crash  prevention,  and  improve  vehicle  safety  (Yih  &  Gerdes,  2005).  These 
successes  support  the  idea  that  autonomous  mobility  and  related  technologies  can  be  integrated  into 
military  platforms  to  augment  the  driver's  capabilities  and  improve  overall  performance. 

Autonomous  mobility  in  manned/unmanned  ground  vehicles  can  be  integrated  in  several 
manners.  The  system  could  be  designed  to  provide  information  about  potential  routes,  limit  control 
inputs  from  the  human  operator  or  operate  in  a  collaborative  scheme  to  leverage  desirable  control 
capabilities  from  both  human  and  nonhuman  sources  (Crandall  et  al,  2005;  Fong  et  al,  2001;  2003). 
Supervisory  control  has  potentially  the  greatest  impact  on  manned  vehicle  operations  and  has  had  success 
enabling  vehicle  mobility  and  reducing  operator  workload  for  variety  of  unmanned  vehicles  in  other 
hazardous  environments  (i.e.  space  exploration,  search  and  rescue  following  the  collapse  of  the  WTC  on 
9/11).  However,  Stanton  &  Young  (1998)  argued  that  integrating  autonomous  components  into  a  system 
does  not  necessarily  reduce  workload;  for  example,  automation  can,  in  certain  cases,  cause  problems  with 
reclaiming  vehicle  control.  Moreover,  it  has  been  argued  that  automation  does  not  remove  tasks  from  the 
operator  but  rather,  it  changes  the  operator's  responsibilities.  This  shift  in  operator  responsibilities  has 
both  costs  and  benefits  (Parasuraman  &  Riley,  1997).  Therefore,  despite  existence  of  various  methods  for 
integrating  autonomous  technologies  into  manned/unmanned  vehicles,  problems  remain  in  terms  of 
resulting  effects  on  Soldier-vehicle  system  performance.  We  will  be  examining  the  impact  of  various 
mobility  technologies  on  driving  performance. 

To  address  these  technology  concerns,  six  different  autonomous  planning  driving  aids  were 
developed:  Confidence,  Behavior  Levels,  Steerable  Waypoint,  GPS  Adjust,  Safety  Push/Clear  Map,  and 
Cost  Map.  The  confidence  driving  aid  is  a  colored  indicator  that  describes  the  complexity  of  the  path, 
speed,  cost  of  the  path,  and  environment.  The  behavior  levels  driving  aid  is  a  visual  and  changeable  value 
to  increase/decrease  the  autonomy’s  maneuvering  intensity  in  the  aggressiveness  and  urgency  categories. 
The  steerable  waypoint  driving  aid  allows  the  operator  to  change  the  target  waypoint  by  using  a 
controlling  device.  This  will  allow  the  operator  to  change  the  long  distance  waypoint  while  the  vehicle 
corrects  the  short  waypoints.  The  GPS  adjust  driving  aid  allows  the  ability  such  that  the  operator  can 
shift  the  current  path  by  some  delta  in  Northing  and  Easting  taking  into  consideration  GPS  “pops” 
(instances  where  one  or  more  GPS  satellites  is  lost  causing  a  temporary  discontinuous  jump  in  reported 
vehicle  position  when  no  such  change  has  actually  occurred).  The  GPS  adjust  path  can  be  moved  which 
allows  the  vehicle  to  plan  a  new  path  to  this  goal.  The  safety  push/clear  map  is  a  driving  aid  that  will 
allow  the  operator  to  clear  the  world  model  maps.  The  Cost  Map  driving  aid  displays  the  world  map  to 
the  operator. 

Due  to  experimental  constraint  (e.g.,  experimental  control,  time,  location,  etc),  only  the  steerable 
waypoint  driving  aid  was  selected  for  experimental  evaluation. 

Unmanned  Systems  Mobility  Experiment  Objective 

The  objective  of  the  unmanned  systems  mobility  research  was  to  quantify  the  operator 
performance  with  the  new  vehicle  control  mode  technologies  developed  and  to  identify  technology  areas 
for  future  performance  improvements.  This  experiment  compared  three  mobility  modes  (teleoperation, 
autonomous  mobility  with  teleoperation,  and  autonomous  mobility  with  steerable  waypoint). 
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METHOD 


Participants 

Volunteer  participants  consisted  of  13  active-duty  Soldiers  from  the  2nd  Combined  Arms 
Battalion  5th  Brigade  (Army  Experiment  Task  Force)  located  at  Ft.  Bliss,  TX.  All  of  the  Soldiers  were 
male  ranging  in  age  from  20-3 1  with  an  average  of  24  years  of  age. 

Experiment  Apparatuses 

The  driving  aids  experiment  required  a  manned  platform  to  host  the  participant  and  the  robotic 
monitoring  and  control  systems,  and  an  unmanned  platform  to  act  as  the  robotic  asset.  The  manned 
platform  utilized  was  the  Crew-integration  and  Automation  Testbed  (CAT  -  modified  prototype  Stryker 
vehicle),  and  the  unmanned  platform  was  the  Experimental  Unmanned  Vehicle  (XUV);  Figure  1  provides 
images  of  both  vehicles. 

Each  vehicle  contained  a  complex  suite  of  electronic  equipment  for  vehicle  control,  sensing, 
information  exchange,  and  data  logging. 


Figure  1.  CAT  and  XUV 

Crew-Automation  and  integration  Testbed  (CAT).  The  manned  platform  used  for  this  experiment 
was  the  Crew-integration  and  Automation  Testbed  (CAT).  The  CAT  is  a  Stryker  that  has  been  modified 
for  technology  integration.  There  are  two  crewstations,  a  Mission  Module  Work  Station  (MMWS)  and  an 
experimenter  station  in  the  vehicle.  Each  station  is  comprised  of  a  seat  for  an  operator  and  a  console 
containing  the  Soldier  Machine  Interfaces  (SMls),  each  of  which  has  video  displays  and  controls  to 
enable  the  operator  to  command  one  or  more  remote  assets  or  the  CAT  itself.  The  CAT  can  carry  up  to  a 
crew  of  five,  including  a  safety  driver,  experimenter/observer,  and  three  operators  (one  at  each  work  or 
crewstation).  During  the  Capstone  experiment,  the  crew  consisted  of  the  participant  seated  at  the  MMWS, 
two  experimenters  seated  at  another  crewstation  or  the  experimenter  station,  and  a  trained  safety  driver 
who  was  responsible  for  CAT  mobility  and  the  safety  of  experimenters  and  participants  inside  the  vehicle. 
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Figure  2.  CAT  Platform  and  the  MMWS 

CAT  Platform  and  the  MMWS.  The  multi-mission  SMls  support  the  fight  (19K),  scout  (19D), 
and  carrier  (1 1M)  Military  Operational  Specialty  (MOSs)  as  well  as  the  command  and  control  of 
unmanned  assets.  These  MOSs  and  the  need  for  unmanned  asset  control  are  the  basis  of  future  force 
manned  ground  vehicle  concept  development.  Each  crewstation  generally  allows  equivalent  capabilities 
and  responsibilities  and  essentially  have  the  same  software  allowing  role-specific  (e.g.  Vehicle 
Commander,  Driver,  or  Robotics  Operator)  assistance.  Each  crewstation  can  be  configured  to  support  a 
specific  role  at  any  given  point  in  time  depending  on  the  mission  and  task  parameters. 


Experimental  Unmanned  Vehicles  (XUV).  The  unmanned  platform  used  in  the  experiment  was 
the  Experimental  Unmanned  Vehicle  (XUV).  The  XUV  was  developed  under  the  Demo  111  program  and 
has  similar  autonomous  mobility  systems  as  the  CAT,  however,  the  XUV  is  smaller,  has  only  4  wheels, 
and  can  also  be  teleoperated  (Figure  ). 


Figure  3.  XUV 

Multi-Mission  Workstation  (MMWS).  The  MMWS  was  the  operational  hardware  platform  used 
by  the  participants,  and  it  was  located  in  the  rear  right  side  of  the  CAT.  It  is  an  FCS  surrogate  mission 
work  station  used  for  robotic  asset  monitoring  and  control.  The  Scalable  Soldier  Machine  Interface 
(SSM1)  software  provided  FCS  style  displays  and  controls,  allowing  participants  to  interface  with  the 
experimental  environment  via  robotic  vehicle  control  devices,  a  video  feed  from  the  XUV  camera,  and  a 
common  operating  picture  map  display. 
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Subjective  Questionnaires.  Subjective  workload  and  situation  awareness  ratings  were  measured 
with  the  NASA-Task  Load  Index  (NASA-TLX,  Hart  &  Staveland,  1987)  and  the  Cognitive  Compatibility 
Situation  Awareness  Technique  Questionnaire  (CC  -  SART,  Taylor,  1990),  respectively.  The  NASA- 
TLX  is  a  multi-dimensional  rating  procedure  that  derives  an  overall  workload  score  based  on  a  weighted 
average  of  ratings  on  six  subscales  (Mental  Demand,  Physical  Demand,  Temporal  Demand,  Own 
Performance,  Effort,  and  Frustration).  The  Cognitive  Compatibility  Situation  Awareness  Technique 
Questionnaire  is  a  subjective  rating  of  situation  awareness.  Participants  rated  their  experience  with  the 
task  on  three  dimensions:  Activation  of  Knowledge,  Ease  of  Reasoning,  and  Level  of  Processing.  Level  of 
Processing  is  the  degree  to  which  the  situation  involves,  at  the  low  level,  natural  automatic,  intuitive,  and 
associated  processing,  or  at  the  high  level,  analytic,  considered,  conceptual  and  abstract  processing.  Ease 
of  Reasoning  is  the  degree  to  which  the  situation,  at  the  low  level,  is  confusing  and  contradictory,  or,  at 
the  high  level,  is  straightforward  and  understandable.  Activation  of  Knowledge  is  the  degree  to  which  the 
situation,  at  the  low  level,  is  strange  and  unusual,  or,  at  the  high  level  is  recognizable  and  familiar. 
Workload  and  situation  awareness  were  assessed  at  the  end  of  each  mission. 

The  Usability  Questionnaire.  The  Usability  questionnaire  was  administered  at  the  conclusion  of 
all  experiment  runs,  collecting  subjective  input  on  the  usability  of  the  SMI  in  general  and  the  steerable 
waypoint.  Responses  to  individual  questions  were  averaged  and  using  the  Likert  scale,  a  percentage  of 
total  possible  (score)  was  calculated.  An  exit  interview  was  also  completed  at  the  end  of  all  experiment 
runs  collecting  participants’  impressions  of  the  system  and  recommendations  for  future  improvement. 

Experimental  Procedures 

The  experiment  began  with  an  introduction  of  the  purpose  of  the  RC  ATO  Capstone  experiment. 
This  was  a  slide  presentation  that  described  why  autonomous  aided  technology  is  emerging  and  how  it 
works  at  a  mechanistic  level.  Each  participant  received  this  presentation  in  the  beginning  of  the  training 
process.  After  the  presentation,  they  received  an  interactive  training  session  on  the  System  Integrated 
Laboratory  (S1L)  located  in  a  closed  facility  at  the  Ft.  Bliss  test  site  (Figure  1).  The  S1L  is  a  replicated 
MMWS  with  the  ability  to  train  the  participants  in  a  classroom  setting.  They  were  able  to  become 
acclimated  to  the  tasks  for  the  experiment,  the  hardware,  software,  technology,  etc.  Participants  were  able 
to  practice  the  mobility  conditions  in  the  S1L  by  driving  a  simulated  XUV.  The  terrain  database  simulated 
the  Fort  Bliss  test  range,  allowing  the  participants  to  become  familiar  with  the  local  terrain. 
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Figure  1.  System  Integrated  Laboratory  (SIL) 

The  SIL  was  established  on  site  at  the  Fort  Bliss  test  range  for  the  dual  purposes  of  conducting 
participant  training  and  also  demonstrating  the  constructive  Soldier  in  the  virtual  environment. 

During  the  experimental  run  (individual  mission)  the  participant  operated  the  XUV  from  the 
MMWS  in  the  CAT  which  was  on  the  move,  across  approximately  3.3  km  at  no  more  than  16  kph  using 
an  indirect  vision  system.  Terrain  at  Ft.  Bliss  consisted  of  consisted  of  arid  desert  roads  with  complex 
sand  dunes  that  had  a  multitude  of  vegetation  growing  on  them  (see  Figure  5). 


Figure  5.  Ft.  Bliss  Terrain 

The  exact  route  of  travel  was  clearly  visible  and  was  provided  to  the  participant.  Temporary 
obstacles  (cones  and  barrels)  were  placed  on  the  planned  route  and  were  set  up  so  as  to  require  the 
participant  to  perform  various  types  of  pre-planned  maneuvers  in  addition  to  the  general  objective  of 
completing  the  course  while  staying  on  the  priori  path.  The  maneuvers  of  interest  included  placing  the 
Soldiers  in  situations  where  they  had  to  alternately  avoid  an  obstacle  that  the  autonomous  mobility  system 
should  detect,  avoid  an  obstacle  that  the  autonomous  mobility  system  should  not  detect,  traverse  a  route 
parallel  to  the  planned  path,  and  stay  on  the  road  while  navigating  a  tight  comer.  The  participant  was 
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instructed  to  traverse  the  XUV  as  quickly  as  possible  along  the  provided  route,  adhering  as  closely  as 
possible  to  the  planned  route,  and  avoiding  the  temporary  obstacles  as  necessary. 

Experimental  Course  Design 

The  test  course  featured  a  series  of  challenges  for  the  operator  and  autonomy  alike.  Several 
obstacles  were  placed  along  the  route  that  would  necessitate  operator  control  of  the  XUV.  Figure  6 
depicts  an  overview  of  the  operational  test  course  at  the  Fort  Bliss,  TX  test  range. 


Figure  6.  Experimental  Test  Course 

The  test  course  featured  ten  planned  events,  or  vehicle  maneuvers,  that  an  operator  faced  during  a 
test  run.  Additionally,  the  participants  were  asked  to  perform  reconnaissance  of  a  designated  area  along 
the  route,  based  on  the  specific  scenario  they  were  executing. 

Several  obstacles  were  created  using  either  cones,  barrels,  or  a  combination  of  both.  Traffic  cones 
represented  obstacles  that  were  too  short  for  the  XUV’s  Autonomous  Mobility  System  to  detect,  thus 
requiring  human  intervention  to  avoid  collisions.  Barrels  were  large  enough  to  be  detected  by  the  XUV 
Autonomous  Mobility  System,  and  thus  were  considered  obstacles  that  the  robot  would  naturally  avoid 
during  its  continuous  route  planning  activities  while  navigating  autonomously.  For  example,  the  cone 
narrows  measured  the  ability  of  the  participant  to  detect  and  avoid  cones  while  following  an  a  priori  route 
that  would  otherwise  cause  collisions  between  the  XUV  and  one  or  more  cones.  The  cone  slalom 
measured  the  ability  of  the  participant  to  maneuver  the  XUV  in  a  weaving  pattern  through  a  series  of 
cones  that  were  spaced  15  meters  apart.  Again,  the  participant  was  responsible  for  obstacle  detection  and 
avoidance  in  all  test  conditions  since  the  cones  were  too  short  for  the  XUV  LADAR  to  sense,  and  the  a 
priori  route  would  otherwise  guide  the  robot  through  the  middle  of  the  slalom  when  navigating 
autonomously.  NA1  zones  were  also  included  to  simulate  real-time  commands  to  perform  reconnaissance 
off  of  the  a  priori  route.  This  tested  the  Soldiers  ability  to  react  to  changes  in  mission  without  a  full 
vehicle  re -planning  exercise.  In  responding  to  a  simulated  target  of  opportunity,  participants  were 
required  to  temporarily  deviate  from  the  planned  route,  read  a  sign  at  one  of  six  designated  areas,  and  then 
return  the  robot  to  the  a  priori  route. 
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Experimental  Secondary  Task 

During  each  trial,  the  participant  received  queries  over  headphones  connected  to  the  intra-vehicle 
audio  mix.  These  queries  assessed  the  participant’s  situational  awareness  of  their  environment.  They 
received  a  query  approximately  once  per  minute,  or  approximately  12  per  trial.  To  try  and  capture  how 
much  mental  effort  it  took  to  navigate  the  obstacles  in  the  various  cases  we  asked  a  total  of  12  questions. 
Six  of  the  questions  were  asked  in  an  obstacle,  and  the  other  six  were  asked  in  the  space  between 
obstacles.  A  number  of  questions  were  generated  such  that  no  one  question  appeared  in  every  scenario. 

The  queries  assessed  level  1  (perception),  level  2  (comprehension),  and  level  3  (projection) 
situation  awareness.  The  participant  responded  to  the  queries  through  the  radio.  Below  are  examples  of 
the  different  levels  of  situational  awareness  questions  that  were  asked  to  the  participants. 

Level  1  (perception):  “What  is  the  current  heading  of  the  XUV?” 

Level  2  (comprehension):  “In  which  direction  did  you  last  deviate  from  your  last  route?” 

Level3  (projection):  “How  long  until  you  reach  NA1  Red?” 

Experimental  Design 

The  experiment  was  a  3x2  within-subject  design.  The  independent  variables  are  mobility 
condition  and  time.  Mobility  condition  had  three  levels:  teleoperation,  autonomous  mobility  with 
teleoperation  intervention,  and  autonomous  mobility  with  steerable  waypoint  intervention.  Thus,  the 
participant  completed  six  missions.  The  second  independent  variable  was  time.  Each  mobility  condition 
was  repeated  two  times.  During  the  experiment,  the  participant  controlled  the  XUV  with  the  MMWS. 

Teleoperation  (TEL).  In  the  teleoperation  mode,  the  participant  controlled  the  XUV  through 
direct  control  in  the  MMWS  via  the  joystick  as  their  steering  device,  and  an  accelerator  and  a  brake  for 
speed  control.  Participants  directly  controlled  the  XUV  throughout  the  mission  in  this  mode. 

Autonomous  Mobility  with  Teleoperation  (ATL).  In  the  ATL  mode,  the  participant  controlled  the 
XUV  through  Autonomous  Mobility.  The  XUV  was  provided  an  a  priori  route  plan  to  complete  a  route 
through  the  experimental  test  course.  Once  the  mission  was  executed  the  XUV  moved  through  the  terrain 
without  requiring  direct  control.  When  an  obstacle  was  encountered  the  participant  was  expected  to 
release  the  XUV  from  the  AM  Mode  into  teleoperation  mode  and  use  the  direct  controls  to  maneuver 
through  the  obstacles.  Control  of  the  XUV  was  then  returned  to  the  AM  mode. 

Autonomous  Mobility  with  Steerable  Waypoint  (ASW).  In  the  ASW  mode,  the  participant 
controlled  the  vehicle  in  a  manner  similar  to  the  ATL  condition,  but  using  the  Steerable  Waypoint 
intervention  method  instead  of  teleoperation.  The  Steerable  Waypoint  driving  aid  is  a  technology  that  was 
developed  to  allow  the  robotic  operator  to  seamlessly  intervene  in  control  that  would  otherwise  be 
handled  by  the  autonomous  mobility  system.  Steerable  waypoint  allows  the  operator  to  change  (or  steer) 
the  near-term  goal  of  the  autonomy  in  real-time,  rather  than  requiring  the  XUV  to  be  stopped  and 
switched  into  full  teleoperation  mode  (an  operation  that  would  have  to  be  done  in  reverse  once  the 
intervention  was  complete).  The  waypoint  provided  when  the  ASW  mode  is  engaged  serves  as  the  goal 
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for  the  autonomous  system,  essentially  acting  as  a  “carrot  on  a  stick”  and  thus  replacing  the  long-range 
planning  point  currently  being  used  by  the  AM  system.  If  the  operator  places  the  waypoint  directly  in 
front  of  the  vehicle,  the  vehicle  attempts  to  go  straight.  As  the  operator  swings  the  point  to  one  side,  the 
vehicle  attempts  to  turn  towards  that  side.  When  engaged  in  steerable  waypoint,  the  operator  maintains 
the  ability  to  control  vehicle  mobility  by  instantaneously  changing  a  target  waypoint  distance  from  the 
vehicle  (20m  to  60m).  However,  worthwhile  to  note  is  the  ASW  mode  differs  in  an  important  manner 
from  direct  teleoperation.  When  using  the  steerable  waypoint,  the  AM  system  remains  engaged  and 
continues  to  calculate  and  select  the  specific  path  taken  between  the  current  and  specified  location  of  the 
XUV.  As  such,  the  ASW  mode  was  intended  to  have  the  additional  advantage  of  utilizing  the 
safeguarding  aspects  of  the  AM  system  while  providing  the  operator  a  greater  sense  of  control  over  the 
vehicle.  In  essence,  this  driving  aid  was  intended  to  be  an  advanced  version  of  guarded  teleoperation. 
Figure  illustrates  the  steerable  waypoint  indicator  on  the  interface.  The  lower  display  is  the  XUV’s 
video;  the  steerable  waypoint  overlay  is  the  green  cones  and  lines.  The  more  transparent  green  cone  is  the 
reference  point  (starting  location).  The  brighter  green  cone  is  the  location  of  where  the  operator  is 
“steering”  the  waypoint.  The  lines  are  indicators  for  direction  from  the  reference  point. 


Figure  7.  Steerable  Waypoint  Driving  Aid 


Dependent  Measures 

Objective  Performance: 

To  provide  the  most  options  for  data  analysis,  all  vehicle  control  variables  were  calculated  within 
‘performance  zones’  reflecting  a  premeditated  set  of  maneuvers  designed  to  assess  both  advantages  and 
disadvantages  of  the  three  vehicle  control  methods  (mobility  conditions).  Route  conformance  was 
calculated  from  the  straight-line  differences  between  the  experimentally-observed  vehicle  position  and  an 
'ideal'  trajectory.  The  ideal  trajectory  was  determined  by  having  an  expert  operator  teleoperate  the  XUV 
through  a  complete  run  of  the  experimental  course  while  navigating  all  obstacles  except  for  the  NAls. 
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The  absolute  values  of  the  deviation  scores  were  averaged  within  each  prescribed  performance  zone  and 
were  later  aggregated  for  statistical  analysis.  Because  the  speed  of  the  vehicle  and  the  size  of  each  zone 
varied,  this  measure  of  route  conformance  was  based  on  a  different  number  of  observations  within  each 
zone  than  within  the  other  zones  on  the  course.  As  such,  the  overall  route  deviation  was  based  on  a 
weighted  mean  across  all  performance  zones  within  participant,  run  and  condition,  where  the  number  of 
data  points  contributing  to  each  mean  was  used  as  the  weighting  variable. 

•  Route  Deviation  as  measured  in  mean  meters  the  XUV  vehicle  deviated  from  an  ideally-driven 
route  over  the  experimental  course. 

•  Average  speed  calculated  by  taking  the  average  of  the  speed  over  an  area. 

•  Mission  Time  calculated  as  the  time  elapsed  while  the  XUV  was  on  the  course;  defined  as  the  first 
and  last  points  in  time  when  the  vehicle  was  detected  within  pre-established  start  and  end 
boundaries  based  on  GPS  coordinates. 

•  Number  of  interventions  with  the  XUV  counted  as  the  number  of  times  the  Soldier  assumed 
control  of  the  XUV  during  the  experimental  run  following  a  period  of  time  where  the  AM  system 
was  in  control  (thus,  for  teleop  runs,  there  was  only  a  single  intervention  for  all  participants  as  the 
Soldier  was  in  control  for  the  entire  time). 

•  Time  required  to  intervene  with  the  XUV  calculated  as  the  time  elapsed  while  the  Soldier 
maintained  control  of  the  XUV  during  an  intervention  epoch,  defined  as  the  moment  the  Soldier 
assumed  control  until  the  moment  when  he  returned  control  to  the  AM  system. 

Subjective  Performance: 

•  Subjective  Workload  was  assessed  following  each  mission  (experimental  run)  using  the  NASA- 
TLX  as  described  above  (section  2. 1 . 1 . 1 .4) 

•  Situation  Awareness  was  probed  throughout  each  mission  as  described  in  greater  detail  below 
(section  2. 1.1. 4.6) 

•  Interface  Usability  Survey,  as  described  above  (sees  section  0),  was  administered  after  the  Soldier 
had  completed  all  experimental  runs. 

RESULTS 

Objective  Performance 

Means  and  standard  errors  were  calculated  for  each  dependent  variable.  Mixed  linear  model 
analyses  were  conducted  to  examine  the  effects  of  level  of  mobility  on  objective  performance,  workload, 
and  situation  awareness. 

Mobility  Performance 

Results  showed  that  level  of  automation  significantly  impacted  various  facets  of  mobility 
performance.  Missions  were  completed  faster  in  the  TEL  mode  than  both  ATL  and  ASW. 

Mixed  linear  model  analyses  revealed  a  significant  main  effect  of  level  of  mobility  on  time  to 
complete  a  mission,  F  (2,  32.93)  =  86.88,  p<.00.  Paired  comparisons  revealed  that  this  main  effect  was 
due  to  the  difference  in  completion  times  between  the  teleop  and  two  Autonomous  conditions,  but  the  two 
Autonomous  conditions  did  not  differ  between  themselves,  teleoperation  mode  (1004.16  +  27.84sec)  was 
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faster  than  the  other  modes  and  there  was  no  difference  in  time  between  ATL  (1425.50  +  24.50  sec)  and 
ASW  (1471.36  ±49.08sec),  ps<.  01  and  ps  >.56. 

Mobility  Performance  within  Obstacles 

Mobility  performance  was  averaged  across  obstacles.  Overall  mobility  performance  varied  across 
mobility  level.  More  specifically,  obstacles  were  completed  faster  in  the  TEL  mode  than  both  the  ATL 
and  ASW.  However,  halt  time  was  less  in  the  ASW  mode  than  TEL.  In  addition,  route  deviation  was 
highest  in  the  ASW  mode. 

Mixed  linear  model  analyses  revealed  a  significant  main  effect  of  level  of  mobility  on  time  to 
complete  the  obstacle,  F  (2,  33.73)  =  26.06  ,  p<.  00.  Additional  model  analyses  for  halt  time  and  route 
deviation  showed  a  significant  main  effect  for  mobility  level,  F(2,18)  =  14.33  ,p<.00  and  F  (2,  27.25)  = 
9.29  ,  p<.  001,  respectively.  To  resolve  the  interactions  paired  comparisons  were  conducted  and  showed 
that  all  the  mobility  modes  were  significantly  different  from  each  other  for  time  to  complete  an  obstacle, 
halt  time,  and  route  deviation,  ,  ps<.00.  See  Table  1  for  the  mean  and  standard  errors  associated  with 
these  effects. 


Table  1.  Mean  (Standard  Error)  Mobility  Performance  within  Obstacles 


TEL 

ATL 

ASW 

Total  Time  (sec) 

19.59(.73) 

26.40(1.34) 

30.82(1.53) 

Halt  Time  (sec) 

.02  (.02) 

2.78(.52) 

.73  (.32) 

Route  Deviation  (m) 

1.21  (.09) 

1.21  (.05) 

2.25(.24) 

Table  2  explains  the  percent  of  time  the  robotic  operator  had  to  control  the  driving  task  of  the  robot.  The 
autonomous  mobility  decreases  the  task  of  driving  by  45%  in  both  situations. 


Table  2.Percent  of  Time  in  Mobility  Modality 


Mobility  Condition 

Teleoperation 

Autonomous  Mobility 

Steerable  Waypoint 

TEL 

100% 

- 

ATL 

53.1% 

46.8% 

- 

ASW 

- 

45.2% 

56.1% 

Mobility  Performance  within  the  NAI 

Mobility  performance  was  analyzed  within  each  NAI.  Overall  mobility  performance  varied  across 
mobility  level.  More  specifically,  NAI  were  completed  faster  in  the  TEL  mode  than  both  the  ATL  and 
ASW.  In  addition,  halt  times  were  shortest  in  TEL  mode.  Halt  times  in  ASW  were  longer  than  TEL  but 
shorter  than  ATL. 

Mixed  linear  model  analyses  revealed  a  significant  main  effect  of  level  of  mobility  on  time  to 
complete  the  NAI,  F  (2,  30.32)  =  1 0.08,  p<.0().  To  resolve  the  interaction  paired  comparisons  were 
conducted.  Time  to  complete  the  NAI  was  faster  in  TEL  than  ATL  or  ASW  mode,  p<.  00.  Additional 
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model  analyses  for  halt  time  showed  a  significant  main  effect  for  mobility  level,  F( 2,  33.93)  =  18.16,  p<. 
00.  Paired  comparisons  revealed  that  halt  time  was  significantly  shorter  in  TEL  than  AI  L,  p<.  00. 

Further,  halt  time  was  significantly  shorter  in  ASW  than  ATL ,p<.  00  (see  Table  3). 


Table  3.  Mean  (Standard  Error)  Mobility  Performance  within  the  NAIs 


Mobility  Condition 

TEL 

ATL 

ASW 

Overall 

Time(sec) 

183.59  (14.60) 

226.36(15.75) 

258.69  (29.43) 

Halt  Time  (sec) 

3.01  (1.75) 

27.25  (4.10) 

10.34  (5.09) 

Secondary  Task  performance:  Situation  Awareness  (SA)  Queries 

Two  aspects  of  secondary  task  performance  were  analyzed,  reaction  time  (seconds)  and  accuracy 
for  each  level  of  SA.  Results  for  level  one  and  two  SA  queries  showed  that  level  of  mobility  did  not 
significantly  affect  performance  (reaction  time  or  accuracy)  on  the  secondary  task.  However,  performance 
on  level  three  SA  queries  was  significantly  affected  by  level  of  automation.  Mixed  linear  model  analyses 
revealed  a  significant  main  effect  of  level  of  mobility  for  reaction  time  to  level  three  SA  queries  ,  F  (2, 
20.87)  =  3.58,  p<.04.  To  resolve  the  interaction  paired  comparisons  were  conducted.  Responses  to  SA 
queries  was  faster  in  TEL  than  ASW,  p<.  02.  Further,  responses  were  faster  in  ATL  mode  than  ASW,  p<. 
02.  Though  not  significant  (p=.09),  a  similar  pattern  of  result  occurred  for  accuracy  on  the  level  three  SA 
queries.  Accuracy  was  higher  in  TEL  mode  than  ASW  (see  Table  4) . 


Table  4.  Mean  (Standard  Error)  Performance  for  SA  Queries 


TEL 

ATL 

ASW 

Reaction  Time  (sec) 

Level  1 

7.38  (.91) 

9.41  (.97) 

8.99  (1.34) 

Level  2 

8.18  (1.01) 

9.62  (.95) 

9.32(1. .7) 

Level  3 

11.30  (1.46) 

8.79  (.57) 

12.78  (1.48) 

Accuracy  Score  (na) 

Level  1 

.83  (.03) 

.81  (.04) 

.76  (.05) 

Level  2 

.74  (.04) 

.73  (.05) 

.69  (.05) 

Level  3 

.79  (.03) 

.69  (.06) 

.68  (.03) 

Subjective  Performance 
Workload 

Results  for  subjective  workload  showed  that  perceived  performance  was  poorer  and  frustration 
was  higher  in  ASW  than  TEL  mode.  Mixed  linear  model  analyses  revealed  a  significant  main  effect  of 
level  of  mobility  for  performance  and  frustration,  F  (2,  39.50)  =  7.24,  p<.00  and  F  (2,  33.75)  =  4.72, 


13 


/;<.()  1 ,  respectively.  To  resolve  the  interaction  paired  comparisons  were  conducted.  Performance  was 
lower  in  ASW  than  TEL  mode  and  ATL  mode,  ps<.  01.  Similarly  frustration  was  higher  in  ASW  than 
TEL  mode  and  ATL  mode,  ps<.  01.  There  was  no  significant  difference  between  levels  of  mobility  for 
the  other  dimensions  of  workload  (See  figure  2). 


Mental  Physical  Temporal  Performance  Effort  Frustration 

Workload 


Figure  2.  Workload  Performance 

There  were  no  significant  differences  between  the  levels  of  subjective  situation  awareness, 
reported  by  the  CC-SART,  between  the  levels  of  automation.  The  lack  of  significant  result  may  have  been 
due  to  a  lack  of  understanding  of  the  levels  of  SA  this  scale  by  the  participants. 


14 


Soldier  Feedback 


The  Usability  questionnaire  was  administered  at  the  conclusion  of  the  experiment,  collecting 
subjective  input. 


Steerable  Waypoint 
Usability 


Easy  to  Use 


System  Control  System  Feedback  Supports  Mobility  Task 


Figure  9.  Steerable  Waypoint  Usability  Responses 

While  the  Soldiers  responded  positively  to  the  steerable  waypoint  concept,  the  implementation 
tested  was  not  regarded  as  a  useful  technology,  especially  when  compared  to  autonomy  intervention  with 
teleoperation  (See  Figure  9).  The  following  exit  interview  summary  captures  feedback  on  the  steerable 
waypoint. 

Exit  Interview  Responses:  The  exit  interview  was  completed  with  each  Soldier  at  the  conclusion 
of  all  experimental  runs  in  an  effort  to  collect  impressions  of  the  system  and  recommendations  for  future 
improvement.  The  exit  interview  specifically  addressed  the  following  topics: 

•  Autonomous  Mobility  System  (AMS)  performance 

•  Steerable  waypoint  driving  aid  performance 

•  Teleoperation  system  performance 

•  Preferences  for  vehicle  control 

The  Soldier  rated  overall  performance  of  the  AMS  a  seven  on  a  scale  of  one  to  ten,  where  ten  is 
best.  They  would  all  use  this  system  in  the  field,  indicating  that  it  allowed  them  to  multi-task,  focusing 
on  other  critical  mission  objectives,  such  as  searching  for  Improvised  Explosive  Devices  (IEDs)  along  the 
route. 


Strengths  included  freeing  up  time/personnel  to  complete  other  tasks,  “ease  of  mind”  that  the 
vehicle  would  stay  on  course  while  taking  the  safest  route,  and  the  system’s  ability  to  make  its  own 
decisions  to  avoid  obstacles.  The  Soldier  felt  the  autonomy  would  be  most  useful  for  missions  with  “a  lot 
of  distance  to  cover”  such  as  convoy  operations  or  as  a  reconnaissance  asset  to  obtain  situation  awareness 
of  an  area  ahead  of  Soldier. 

The  Soldiers  expressed  some  concern  about  the  autonomy’s  judgment  of  and  reaction  to  obstacles 
and  the  impact  course  deviations  may  have  on  vehicle  stealthiness.  They  often  have  to  make  the  decision 
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to  go  over  something  the  autonomy  would  likely  attempt  to  avoid  in  order  to  maintain  a  secure/covert 
route.  Large  obstacles  seemed  to  result  in  the  vehicle  deviating  further  from  the  route,  “which  may  give 
away  its  position  and  require  a  lot  of  oversight.”  The  Soldiers  also  had  specific  concerns  about 
unexpected  autonomy  behaviors,  performance  in  urban  environments,  and  reaction  to  water  as  “water  is 
not  seen  as  an  obstacle.” 

Steerable  waypoint  performance  was  rated  a  four  on  a  scale  of  one  to  ten.  Four  out  of  thirteen 
Soldiers  stated  they  would  use  the  technology  in  a  mission  “if  it  was  fine-tuned  and  I  didn’t  have  to 
sacrifice  control.”  The  steerable  waypoint  provided  a  quick  intervention,  smoother  than  the  teleoperation 
transition  and  allowed  “tighter  AMS  maneuvers.”  It  allowed  the  controller  to  influence  the  autonomy  to 
make  tighter  turns,  nudge  the  vehicle  back  on  path  if  it  strayed,  and  control  autonomy  speed. 

The  Soldiers  used  the  steerable  waypoint  to  steer  the  vehicle  off  course  without  replanning  and  to 
avoid  obstacles  they  thought  the  autonomy  should  go  through,  either  based  on  their  assessment  of  vehicle 
capability  or  for  concealment  considerations. 

Overall,  Soldiers  thought  the  steerable  waypoint  concept  was  good  “if  it  worked.”  They 
described  “fighting  the  hot,”  commenting  that  it  “doesn’t  do  what  you  want”  making  it  “hard  to  navigate.” 
All  of  this  required  too  much  concentration  and  time  as  the  vehicle  seemed  to  stop  more  often  to  “think” 
when  the  steerable  waypoint  was  used.  Soldiers  thought  the  driving  aid  was  more  difficult  to  use  and  less 
accurate  than  teleoperation,  which  outweighed  the  benefit  of  a  quicker  transition. 

Teleoperation  system  performance  was  rated  a  nine  on  a  scale  of  one  to  ten  and  all  thirteen 
Soldiers  would  use  the  system  in  the  field.  Strengths  included  complete  and  reliable  control  of  the  vehicle 
(speed,  heading,  obstacle  avoidance,  maneuvers,  driving  aggressiveness),  ease  of  use,  and  faster  mobility 
than  the  autonomy.  The  Soldiers  did  not  perceive  latency  in  the  control  system  as  they  did  with  the  small 
unmanned  ground  vehicle  (SUGV)  that  they  controlled  in  another  Capstone  test.  Teleoperation  would  be 
most  useful  for  missions  requiring  cover  and  concealment  and  in  situations  where  there  is  not  adequate 
time  to  create  a  route  plan  for  the  autonomy. 

While  teleoperation  was  the  preferred  AMS  intervention  method  for  these  test  scenarios,  the 
Soldiers  did  not  think  that  as  an  individual  controller  they  could  adequately  multi-task  to  complete  other 
mission  objectives.  They  would  also  like  to  see  a  more  seamless  transition  from  autonomy  to 
teleoperation  that  does  not  require  many  steps  or  the  vehicle  to  stop.  Additional  recommended 
teleoperation  improvements  included  a  gimbal  (pan/tilt)  camera  mounted  on  the  vehicle  centerline  with 
zoom  control  and  a  larger  field  of  view  or  more  cameras  placed  on  the  vehicle.  The  Soldiers  also 
requested  a  yoke  replacement  for  the  joystick. 

CONCLUSIONS 

Overall,  there  are  beneficial  qualities  that  autonomy  can  bring  to  Soldier’s  performance  in  respect 
to  workload  and  reducing  interaction  time  with  an  unmanned  system.  Technology  was  developed  to 
allow  for  intervention  methodologies  that  create  a  safe  and  easy  transition  of  control  in  the  instances  that 
autonomy  fails.  Steerable  Waypoint  was  the  technology  developed  to  do  such.  In  this  experiment,  the 
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Soldiers  completed  a  reconnaissance  mission  using  three  modes  of  autonomy.  They  were  responsible  for 
driving  the  robot  and  maintaining  SA  of  the  mission  through  embedded  communications.  In  general,  they 
were  able  to  drive  the  XUV  in  all  the  modes  tested  with  minimal  training  time. 

Though  the  intervention  of  steerable  waypoint  was  not  statistically  better  than  teleoperation  or  the 
teleoperation  intervention,  the  potential  benefits  of  an  intervening  technology  was  high.  The  Soldiers 
requested  an  easy  transition  between  autonomy  and  intervention.  The  teleoperation  intervention,  while 
effective  and  easy  to  use,  required  the  operator  to  stop  the  vehicle,  as  evidenced  by  the  higher  halt  times 
in  teleoperation  mode  than  any  other  mode.  The  Soldiers  requested  the  ability  to  transition  between 
teleoperation  and  autonomy  without  stopping  the  vehicle,  which  they  felt  would  replace  the  need  for  and 
potential  benefits  of  an  improved  steerable  waypoint.  Ten  of  the  thirteen  Soldiers  thought  it  was  easier  to 
intervene  with  teleoperation  than  the  steerable  waypoint  due  to  a  “lack  of  vehicle  response”  to  the  driving 
aid.  Teleoperation  provided  more  control  and  while  the  transition  to  teleoperation  was  “a  bit  longer, 
overall  the  intervention  was  easier.”  If  the  implementation  of  the  steerable  waypoint  was  improved,  we 
are  likely  to  see  a  significant  advantage  in  mobility  and  intervention  performance. 

As  with  all  current  and  future  military  systems,  training  is  essential.  In  this  experiment,  the 
training  time  that  the  participants  received  was  not  sufficient  for  them  to  fully  understand  the  critical 
characteristics  of  the  autonomous  system  and  the  driving  aid  that  was  developed.  They  received  a  half  of 
day’s  worth  of  classroom  and  in-vehicle  training.  In  an  actual  training  session,  for  a  Soldier  to  become 
accustomed  to  a  new  system,  they  will  spend  multiple  weeks  in  a  classroom  with  hands  on  training  before 
expecting  to  use  the  system  in  a  mission  style  environment.  The  lack  of  training  was  a  major  contributor 
to  the  decline  in  the  participant’s  performance  with  the  new  system,  relative  to  the  baseline  technologies. 
The  participants  got  confused  about  the  autonomous  vehicle’s  actions  when  unexpected  events  occurred. 
They  did  not  fully  understand  how  the  autonomous  system  worked  nor  did  they  grasp  the  concept  of 
steerable  waypoint.  This  was  evident  in  the  descriptions  that  were  given  in  the  exit  interviews.  The 
participants  reported  that  in  steerable  waypoint  there  was  a  “lack  of  vehicle  response”.  This  implies  that 
the  participants  did  not  have  a  clear  understanding  on  how  the  steerable  waypoint  worked.  Steerable 
waypoint  is  designed  to  allow  for  an  operator  to  steer  a  waypoint  to  a  location  (20m  to  60m  in  away  from 
the  vehicle)  and  have  the  AMS  do  the  work  to  get  there.  It  may  react  differently  than  a  human’s  desired 
path  to  get  to  the  same  location,  hence  the  cause  of  confusion  and  frustration  levels  that  were  reported  in 
the  subjective  workload  data. 

Another  concern  with  automation  (e.g.,  unmanned  systems)  is  the  operator’s  level  of  trust  and  the 
resultant  level  of  use  of  that  automation.  A  lack  of  trust  in  the  autonomy  may  be  a  contributing  factor  as 
to  why  the  steerable  waypoint  driving  aid  did  not  perform  as  expected.  All  of  the  Soldiers  had  a  MOS  of 
19K;  more  specifically  drivers.  Their  background  in  driving  (Ml  Abrams  Tanks)  was  a  cause  for  concern 
because  of  the  very  specific  duties  that  came  along  with  tank  drivers.  .  They  were  trained  to  be  in  direct 
control  of  their  vehicle  at  all  times.  This  was  their  number  one  priority  in  the  mission.  There  was  a 
perceived  lack  of  control  when  the  vehicle  was  in  AM  mode.  As  such,  they  were  confident  in  the  robots 
location  and  mission  status.  They  were  knowledgeable  about  the  robots  status  and  were  able  to  project 
about  the  mission  more  effectively  in  the  teleoperation  and  autonomy  with  teleoperation.  Research  has 
shown  that  while  there  were  circumstances  where  humans  overly  rely  on  automation,  there  were  other 
equally  important  instances  where  they  should  have  relied  on  automation  and  did  not  (see  Parasuraman, 
2000;  Lee  and  See,  2003).  There  are  two  potential  strategies  that  can  be  used  to  mitigate  this  trust  issue: 
Recruit  participants  with  unmanned  system  or  more  mission  relevant  experience.  Provide  more  in  depth 
training  to  develop  the  operator’s  trust  in  the  system. 

A  secondary  task,  S A  queries,  was  implemented  in  experiment  1 .  Level  3  SA  data  did  not  report 
an  increase  in  AM  modes  compared  to  teleoperation.  Although  autonomy  allows  the  operator  to  attend 
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and  intervene  with  the  robot  less,  there  is  a  cost  in  awareness.  In  teleoperation  the  operator  is  constantly 
engaged  with  the  robot  so  when  queries  about  robot  state  were  asked,  the  queries  required  less  effort  to 
answer.  In  AM  modes,  the  operator’s  attention  is  not  constantly  on  the  robot  and  so  when  queries  were 
asked  that  were  mostly  vehicle  related,  the  operator  needed  to  scan  the  display  more  intensely  for  the 
relevant  information  to  answer  the  query.  This  data  suggests  that  autonomy  does  have  a  cost  but  this  cost 
can  be  remedied  by  having  the  important  information  that  the  operator  needs  to  be  made  more  salient. 

This  data  suggests  that  various  modes  of  autonomy  need  to  be  available  to  the  operator.  There  is 
no  one  mode  that  alleviates  the  operator  from  his  driving  that  is  perfect  in  all  conditions.  A  mobility 
system  that  has  multiple  options  and  is  able  to  provide  recommendations  to  the  user  on  what  mode  to  use 
in  the  current  terrain  may  be  the  best  solution.  In  experiment  1 ,  the  Soldiers  had  minimal  understanding  of 
why  they  needed  to  use  one  mode  versus  another  even  though  they  were  trained.  The  experimenters 
provided  cues  to  the  participants  during  this  mission  on  what  mode  to  use.  The  higher  levels  of 
workload  in  the  AM  with  steerable  waypoint  than  teleoperation  may  be  due  to  the  operator’s  frustration 
with  this  mode.  The  lack  of  understanding  of  how  the  technology  worked  as  well  as  how  it  was 
implemented  were  contributors  to  the  workload  levels  and  performance. 

In  conclusion,  autonomous  systems  have  many  beneficial  aspects  for  both  manned  and  unmanned 
systems.  Technology  is  currently  developed  that  allows  an  operator  to  intervene  when  needed  at  a  faster 
and  easier  method.  Further  research  of  enhanced  methodology  of  this  technology  will  generate  increased 
performance  and  likeliness. 


Disclaimer:  Reference  herein  to  any  specific  commercial  company,  product,  process,  or  service  by  trade 
name,  trademark,  manufacturer,  or  otherwise  does  not  necessarily  constitute  or  imply  its  endorsement, 
recommendation,  or  favoring  by  the  United  States  Government  of  the  Department  of  the  Army  (DoA). 
The  opinions  of  the  authors  expressed  herein  do  not  necessarily  state  or  reflect  those  of  the  United 
States  Government  of  the  DoA,  and  shall  not  be  used  for  advertising  or  product  endorsement  purposes. 
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